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DETAILED ACTION 

1 . This office action is in response to the Request for Continued Examination filed 
on 27 January 2009 and the Information Disclosure Statements filed on 27 January 
2009 and 27 February 2009. 

2. Claims 1 , 3, 5, 1 1 -1 4, 21 , 23, 26, 28, 34-37, 44-48, 50-53, and 56 have been 
amended. 

3. The objection of the specification has been withdrawn as necessitated by the 
amendment. The rejection of Claims 23, 26-28, 34-38, 41-44, and 50-54 rejected under 
35 U.S.C. 101 has been withdrawn as necessitated by the amendment. The rejection of 
Claims 1 , 3-5, 11,13,15,18, 20-21 , 23, 26-28, 34, 36, 38, 41 , 43-54, and 56 under 35 
U.S.C. 103(a) as being unpatentable over Eintracht et al (US Patent #6,687,878, filed 
3/1 5/1 999) in further in view of Berquist et al (US Patent #5,821 ,931 , patented 

1 0/1 3/1 998) in further view of Schneiderman (US PGPub 2002/0054059 filed on 
1 0/5/2001 , which is cont. of PCT /US01/04963 filed 2/1 6/2001 , which is a non- 
provisional of provisional application filed 4/17/2000, which is a non-provisional of 
provisional application filed 2/18/2000) in further view of Murray et al (US Patent 
#6,597,800, filed 9/22/1999) in further view of Blank (US Patent 5469536, published 
11/21/1 995) in further view of Conlon et al (US Patent 6,41 1 ,31 3, filed 6/1 4/1 999) has 
been withdrawn as necessitated by the amendment. The rejection of Claims 12 and 35 
under 35 U.S.C. 103(a) as being unpatentable over Eintracht et al in further in view of 
Berquist et al in further view of Schneiderman in further view of Murray et al in further 
view of Blank in further view of Conlon et al in further view of Takaha (US Patent 
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#6,021 ,221 , patented 2/1/2000) has been withdrawn as necessitated by the 
amendment. The rejection of Claims 14 and 37 under 35 U.S.C. 103(a) as being 
unpatentable over Eintracht et al in further in view of Berquist et al in further view of 
Schneiderman in further view of Murray et al in further view of Blank in further view of 
Conlon et al in further view of in further view of Doyle (US Patent #6,616,701 , filed 
4/3/2001 ; continuation of appl #09/316,496, filed 5/21/1999; provisional appl. 
#60/086,620, filed 5/23/1998) has been withdrawn as necessitated by the amendment. 
The rejection of Claims 19 and 42 under 35 U.S.C. 103(a) as being unpatentable over 
Eintracht et al in further in view of Berquist et al in further view of Schneiderman in 
further view of Murray et al in further view of Blank in further view of Conlon et al in 
further view of Balabanovic et al (US Patent #6,976,229, filed 1 2/1 6/1 999). 
has been withdrawn as necessitated by the amendment. 

4. Claims 1 , 3-5, 1 1 -1 5, 1 8-21 , 23, 26-28, 34-38, 41 -54, and 56 are pending. Claims 
1, 23, and 56 are independent claims. 



Information Disclosure Statement 

5. The information disclosure statement (IDS) submitted on 27 January 2009 and 
27 February 2009 is in compliance with the provisions of 37 CFR 1 .97. Accordingly, the 
information disclosure statement is being considered by the examiner. 
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Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 1 , 3-5, 1 1 -1 5, 1 8, 20-21 , 23, 26-28, 34-38, 41 , 43-54, and 56 are rejected 
under 35 U.S.C. 103(a) as being unpatentable over Eintracht et al (US Patent 
#6,687,878, filed 3/1 5/1 999) in further in view of Berquist et al (US Patent #5,821 ,931 , 
patented 10/13/1998) in further view of Schneiderman (US PGPub 2002/0054059 filed 
on 10/5/2001 , which is cont. of PCT /US01/04963 filed 2/16/2001 , which is a non- 
provisional of provisional application filed 4/17/2000, which is a non-provisional of 
provisional application filed 2/18/2000) in further view of Murray et al (US Patent 
#6,597,800, filed 9/22/1999) in further view of Knowlton et al (US Patent 5973692, 
published 1 0/26/1 999) in further view of Conlon et al (US Patent 6,41 1 ,31 3, filed 
6/14/1999) 

As per independent Claim 1, Eintracht et al discloses a method comprising: 
• Displaying the plurality of icons, each of the icons being labelled with one or 
more of the metadata labels with which the icon was associated. (Fig 1B, 1C, 
indicator 16: Discloses a plurality of notes in a location that was 
predetermined placed being attached to various regions on the image 
(Column 7, lines 1 -5) Column 1 5, lines 11-13, e.g., discloses text can be 
inputted which produces the note as a label, thus displayed as a metadata 
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label. In addition, the note itself is the icon, while the text input is the labeling 
aspect of the icon. Thus, Eintracht et al's note is an icon with metadata label 
capabilities of being displayed) 

• Detecting selection of at least one of the displayed plurality of icons. (Column 
15, lines 24-27: Discloses selecting a note to be dragged and dropped to 
another location on an image. The detection is inherently detected by 
positioning the cursor over the note and activating the cursor to note enabling 
it to be moved or dragged to a new location.) 

• linking the one or more metadata labels associated with the selected icon with 
a description of the location of the selected subject within the image, and 
storing the linked one or more metadata labels and the description as an 
annotation of the image. (Eintracht et al discloses in Column 19, lines 42-67, 
notes are stored being associated with a document or image and in FIG 1 1 ; 
Column 17, lines 29-30, Eintracht et al discloses an anchor field for each note 
in a database that stores the coordinates of the of the note in the document 
thus linking the note to the document and disclosing the precise location of 
the note in the document.) 

On the other hand, Eintracht et al fails to specifically disclose displaying the 
image adjacent to said plurality of icons. However, Berquist et al discloses in FIG 4, a 
number of notes being displaying adjacent to an application program executing window 
containing a document. In addition, FIG 6 discloses a note placed adjacent to a window 
containing a document before being dragged onto the document. While Berquist et al 
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discloses an embodiment of notes being adjacent to document, Berquist et al discloses 
a note may be attached to a graphic or a video frame. It was well-known in the art at the 
time of Applicant's invention document have the ability to contain images, or the 
application program running had the ability to only show an image, thus therefore it 
would have been obvious to one of ordinary skill in the art at the time of Applicant's 
invention that a note could be displayed adjacent to the image within a document or 
within an application program since Berquist et al would have provided the benefit to 
Eintracht et al in which preventing objects on the screen from overlapping, and being 
able to determine a location prior in moving the object onto the desired location. 

However, Eintracht and Berquist fail to specifically disclose extracting a plurality 
of metadata labels from an extracting a plurality of metadata labels from an existing 
database of metadata labels to form a list of metadata labels; said metadata labels are 
generated prior to having knowledge of the content of the image. However, 
Schneiderman discloses list of metadata labels, i.e. names of people that appear in a 
selectable menu list wherein the list of names have been previously entered into the 
database. (FIG 3, Paragraph 001 1 , 0015, 0049) Thus, the names were extracted from 
the database and displayed into a selectable list before having knowledge of the content 
of the image appearing next to the list. 

It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have modified Eintracht and Berquist with Schneiderman ability 
to display a list of names labels that were extracted from a database since it would have 
provided the benefit of addresses known problems of annotating commercial and/or 
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personal electronic images, by providing software that permits users to easily 
accomplish such annotation, through the drag-and-drop of annotations from a 
predefined list. 

However, Eintracht, Berquist, and Schneiderman fail to specifically disclose 
detecting a subject within the image, using an image detection method, to form an 
automatically placed bounded region within the image, the automatically placed 
bounded region substantially surrounding the detected subject subject within said 
image. However, Murray et al discloses automatically subjecting the image to primary 
segmentation in which the image is divided up into one or more primary homogenous 
regions each approximating to an object of interest presented by a displayed bounding 
box, wherein each bounding box is the rectangle which encloses the segmented pixels 
forming a primary homogenous region which depicts objects of interest (Column 2, lines 
63-65; Column 4, lines 27-30) Since Murray discloses a method of detecting objects in 
an image and automatically placing bounding boxes around objects of interest, this is 
consider an image detection method. 

It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have combined Eintracht et al, Berquist et al, and 
Schneiderman's method with Murray et al's method since Murray et al's method would 
have provided the benefit of an automatic recognition of subjects to identify the 
presence of people. 

In addition, Eintracht discloses the ability to view, create, modify, or delete notes 
(a form of an icon) (Column 15, lines 10-37). Therefore, one can delete the icon located 
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on the selected subject, and either create a new note or move a note to a new location 
onto that subject. However, Eintracht et al, Berquist et al, Schneiderman's method, and 
Murray et al fail to specifically disclose adjusting the automatically placed bounded 
region by re-sizing, moving or erasing the bounded region based on user input and 
replacing a default icon of the linked one or more metadata labels with an image of the 
selected subject, the image of the selected subject being extracted based on the 
adjusted bounded region. However, Knowlton et al discloses the ability for a user to 
intervene and adjust the source region after Knowlton's method of automatically 
selecting a portion of the original image occurs. (Col 15, lines 31-37; Col 28, lines 17- 
25) Such adjustment methods include re-positioning and re-sizing. (Col 15, lines 50-56) 
In addition, once the area has selected, it is formed into a graphical icon wherein the 
selected area of image from within bounded region is extracted and represented as a 
graphical icon. (Col 10, lines 48-52, Col 11, lines 15-20; Col 14, lines 10-15) Thus, in 
conjunction with Eintracht, a user can delete the note and use the newly created 
graphical icon that includes the extracted portion of the original image; a thus a form of 
replacing. 

It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have combined Eintracht et al, Berquist et al, Schneiderman's 
method, and Murray et al, with Knowlton et al's method since Knowlton et al's method of 
creating graphical icons since have provided the benefit wherein the icons provide 
significantly less space to store and display than do the typical compressed "thumbnail" 



Application/Control Number: 09/826,935 Page 9 

Art Unit: 2178 

images of the prior art, and less network time to communicate from one system to 
another. 

Furthermore, Eintracht et al, Berquist et al, Schneiderman, Murray et al, and 
Knowlton et al fail to specifically disclose dragging the selected icon to the image, such 
that the adjusted bounded region is changed upon the selected icon being dragged over 
the adjusted bounded region, in order to emphasize the adjusted bounded region and 
dropping the selected icon within the adjusted bounded region on, wherein the adjusted 
bounded region corresponds to a selected subject within the image;. However, Conlon 
et al discloses an feature wherein a user using a mouse clicks and drags an icon over a 
(bounded) region for the icon to be dragged, the region is highlighted by the change the 
color of its outline. Thus, when the releases the icon over the region, the icon is dropped 
into the region of subject interest. (FIG 4A, Column 6, line 66 - Col 7, line 20) 

It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have combined Eintracht et al, Berquist et al, Schneiderman's, 
Murray et al, and Knowlton with Conlon et al method's of drop and drag functionality 
involving regions since Conlon et al's method would have provided the benefit to 
informing the user of what region an icon would be dropped in if done so by displaying a 
more clearly graphically indication, such as, highlighting, the region in which the icon is 
hovering over. 

As per dependent Claims 3 and 4, Eintracht et al, Berquist et al, and 
Schneiderman fail to specifically disclose the automatically placed bounded region is 
formed based on an analysis of pixels of the image and the analysis of the colour 
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information of the pixels of the image. However, Murray et al discloses a process in 
which an image is captured by a camera utilizing a two dimensional array of light 
intensity sensitive pixels that carries out the processing on the image data to separate 
and identify objects from the background appearing in the image. (Column 1, lines 15- 
30) Then, Murray et al discloses the bounding box encloses segmented pixels forming 
homogeneous region of objects of interest. (Column 2, lines 63-65; Column 4, lines 27- 
30) Therefore, when objects are being identified, being bounded with boxes and 
separated based on the interest, pixels are being analyzed to determined which pixels 
are the same within the region to be separate from the different region of pixels based 
on the color of the pixel or the difference of visible light shown. 

It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have combined Eintracht et al, Berquist et al, and 
Schneiderman's method with Murray et al's method since Murray et al's method since 
Murray et al's method would have provided the benefit of an automatic recognition of a 
target object that improves the accuracy of target object recognition and identification. 

As per dependent Claims 5, 13, and 48, Eintracht et al, Berquist et al, and 
Schneiderman fail to specifically disclose the automatically placed bounded region 
corresponding to the selected subject is of a predetermined size or determined 
automatically, and the size of the adjusted bounded region corresponding to the 
selected subject is determined based on the analysis. However, Murray et al discloses 
the bounding box just encloses the segmented pixels forming a primary homogenous 
region. (Column 2, lines 63-65; Column 4, lines 27-30) Therefore, only bounding related 
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pixels, the box is determined automatically and is predetermined based on the number 
of related pixels. In addition, when objects are being identified, being bounded with 
boxes and separated based on the interest, pixels are being analyzed to determined 
which pixels are the same within the region to be separate from the different region of 
pixels. Thus, the size is being determined to form a homogenous region. 

It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have combined Eintracht et al, Berquist et al, and 
Schneiderman's method with Murray et al's method since it provided the benefit of only 
bounding a region of related pixels of interest from regions of no interest. 

As per dependent Claim 11, 12, and 1 4, Eintracht et al, Berquist et al, 
Schneiderman, and Murray fail to specifically disclose the step of extracting a part of the 
image based on the bounded region corresponding to the selected subject, displaying 
the extracted part of the image and that the size of the adjusted bounded region is 
changeable by the user in response to receiving a user instruction via the graphical user 
interface. However, Knowlton et al discloses extracting a part of the image based on the 
bounded region corresponding to the selected subject, displaying the extracted part of 
the image and that the size of the adjusted bounded region is changeable by the user in 
response to receiving a user instruction via the graphical user interface. (Col 15, lines 1- 
5; Col 16, lines 52-55; Col 17, 5-10; Col 28, lines 17-25) 

As per dependent Claim 15, Claim 15 recites similar limitations as in Claim 1 and 
is rejected under similar rationale. Furthermore, Eintracht et al discloses: 
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• wherein one or more metadata labels are stored as the annotation of the 
subject and are displayed upon selecting the subject in the image. (Column 7, 
lines 13-17, discloses when a note is selected on the note list, its counterpart 
icon in the document window frame is highlighted) 

As per dependent Claim 18, Eintracht et al discloses a method: 

• wherein said storing step included storing the one or more metadata labels as 
the annotation of the subject of the image by using a tag indicating an 
association with the image (Column 3 Lines 13-36 i.e. the document file for 
storing one or more documents, a notes database located on the server, the 
notes database for storing one or more notes, each note or tag associated 
with a particular document or subject, one or more notes clients coupled to a 
network, each notes client operative to locally display a representation of a 
document remotely stored on the server in the document file, the notes client 
adapted to permit a user to annotate the document with one or more notes, 
the notes client operative to simultaneously display the one or more notes 
associated with the document over the displayed document such that the 
document is viewable along with the one or more notes) 

As per dependent Claim 20, Eintracht et al discloses a method further comprising: 

• the step of e-mailing at least the image to at least one e-mail address based 
on the one or more metadata labels associated with the image. (Column 22 
Lines 33-37 i.e. emails to client or user) 
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As per dependent claim 21 , Claim 21 recites similar limitations as in Claim 1 and 
is similarly rejected under rationale. 

As per independent Claim 23, Claim 23 recites a "computer readable medium..." 
for performing the method of Claim 1 , and therefore is similarly rejected under rationale. 

As per dependent Claim 26, Claim 26 recites similar limitations as in Claim 3 and 
is similarly rejected under rationale. 

As per dependent Claim 27, Claim 27 recites similar limitations as in Claim 4 and 
is similarly rejected under rationale. 

As per dependent Claim 28, Claim 28 recites similar limitations as in Claim 5 and 
is similarly rejected under rationale. 

As per dependent Claim 34, Claim 34 recites similar limitations as in Claim 1 1 
and is similarly rejected under rationale. 

As per dependent Claim 35, Claim 35 recites similar limitations as in Claim 12 
and is similar rejected under rationale. 

As per dependent Claim 36, Claim 36 recites similar limitations as in Claim 13 
and is similarly rejected under rationale. 

As per dependent Claim 37, Claim 37 recites similar limitations as in Claim 14 
and is similar rejected under rationale. 

As per dependent Claim 38, Claim 38 recites similar limitations as in Claim 15 
and is similar rejected under rationale. 

As per dependent Claim 41 , Claim 41 recites similar limitations as in Claim 1 8 
and is similar rejected under rationale. 
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As per dependent Claim 43, Claim 43 recites similar limitations as in Claim 20 
and is similar rejected under rationale. 

As per dependent Claim 44, Claim 44 recites similar limitations as in Claim 21 
and is similar rejected under rationale. As per dependent Claims 45 and 46, Eintracht et 
al, Berquist et al, Schneiderman, and Murray to specifically disclose the description 
includes a location and the size of the adjusted bounded region corresponding to the 
selected subject within the image. However, discloses the ability for a user to intervene 
and adjust the source region after Knowlton et al's method of automatically selecting a 
portion of the original image occurs. The user is provided a mouse controlled "stretchy 
rectangle" used to select a portion of the original image for extraction wherein the 
rectangle may conform to a range of dimensions and aspect ratios. Thus, the size is 
based on the range of dimensions and the location is found by where the rectangle is 
located. (Col 14, lines 49-56, 63-67; Col 15, lines 31-37) 

Thus, it would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have combined Eintracht et al, Berquist et al, and 
Schneiderman, and Murray method with Knowlton et al's method of a bounding box 
since the bounding box data would have provided the user information on the location of 
a area of interest to extract and the size of the area of interest within an image. 

As per dependent Claim 47, Claim 47 recites similar limitations as in Claim 1 and 
is similarly rejected under rationale. 

As per dependent Claim 49, Eintracht et al discloses a method: 
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• the linked one or more metadata labels and the descriptions are stored as an 
annotation of the subject of the image. (Eintracht et al discloses in Column 
19, lines 42-67, notes are stored being associated with a document or image 
and in FIG 11; Column 17, lines 29-30, Eintracht et al discloses an anchor 
field for each note in a database that stores the coordinates of the of the note 
in the document thus linking the note to the document and disclosing the 
precise location of the note in the document.) 
As per dependent Claims 50-51, Claim 50-51 recites similar limitations as in 
Claim 45-46 and is similarly rejected under rationale. 

As per dependent Claims 52, Claim 52 recites similar limitations as in Claim 47 
and is similar rejected under rationale. 

As per dependent claim 53, Claim 53 recites similar limitation as in Claim 14 and 
is similar rejected under rationale. Furthermore, Knowlton et al discloses the size is of 
the adjusted region is based on the user's analysis. (Col 14, lines 62-67; Col 15, lines 
31-37) 

As per dependent Claim 54, Claim 54 recites similar limitations as in Claim 49 
and is similar rejected under rationale. 

As per independent claim 56, Claim 56 recites similar limitations as in Claim 1 
and is similar rejected under rationale. Furthermore, Schneiderman discloses displaying 
a representation of each of the metadata labels in the list. (FIG 3) 
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8. Claims 19 and 42 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Eintracht et al (US Patent #6,687,878, filed 3/15/1999) in further in view of Berquist 
et al (US Patent #5,821 ,931 , patented 1 0/1 3/1 998) in further view of Schneiderman (US 
PGPub 2002/0054059) in further view of Murray et al (US Patent #6,597,800, filed 
9/22/1999) in further view of Knowlton et al (US Patent 5973692, published 10/26/1999) 
in further view of Conlon et al (US Patent 6,41 1 ,31 3, filed 6/1 4/1 999) in further view in 
further view of Balabanovic et al (US Patent #6,976,229, filed 1 2/1 6/1 999). 

As per dependent Claim 19, Eintracht et al, Berquist et al, Schneiderman, Murray 
et al, Blank and Conlon et al fail to specifically disclose the one or more predetermined 
metadata labels associated with the subject of the image are stored in an XML file. 
However, Balabanovic et al discloses metadata regarding information to an image being 
stored in an XML format/file. (FIG 5(a,b)-6; Column 1 0, lines 1 -3, 21 - 22, 40-41 ) 

It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to have combined Eintracht et al, Berquist et al, Schneiderman, 
Murray et al, Knowlton and Conlon et al's method with Balabanovic et al's method of 
using metadata in XML format since it would have provided the benefit of flexibility to for 
the file to be easily translated into other formats to viewed by other on different devices. 

As per dependent Claim 42, Claim 42 recites similar limitations as in Claim 13 
and is similar rejected under rationale. 
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Response to Arguments 

9. Applicant's arguments with respect to claims 1 , 3-5, 1 1 -1 5, 1 8-21 , 23, 26-28, 34- 
38, 41-54, and 56 have been considered but are moot in view of the new ground(s) of 
rejection. 

Arguments address regarding of the new limitations of Claims 1, 23, and 56 
brought forth in "adjusting the automatically placed bounded region by re-sizing, moving 
or erasing the bounded region based on user input and replacing a default icon of the 
linked one or more metadata labels with an image of the selected subject, the image of 
the selected subject being extracted based on the adjusted bounded region" has been 
viewed the new ground of rejection of 35 USC 103(a) under new references using 
Knowlton et al. 

Conclusion 

10. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

• Knowlton et al US Patent 6,181 ,838 filed 4/29/1 999 

• Bates etal US Patent 6,456,307 filed 9/9/1 998 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to David Faber whose telephone number is 571-272-2751. 
The examiner can normally be reached on M-F from 8am to 430pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Stephen Hong, can be reached on 571-272-4124. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 



you have questions on access to the Private PAIR system, contact the Electronic 


Business Center (EBC) at 866-217-9197 (toll-free). 


/David Faber/ 




Examiner, Art Unit 2178 






/CESAR B PAULA/ 




Primary Examiner, Art Unit 2178 



